[Feature] Move response jailbreak detection into a response-stage signal - #3335

…nal (vllm-project#3335)

* [Feature] Move response jailbreak detection into a response-stage signal

Signal types now declare the stage they can be observed at. Every
existing type stays request-stage, so "type:name" and the {type, name}
condition shape are unchanged, and system stats or structured-output
validation would be response-stage types under the same rule.

response_jailbreak is the first of them. Rules are declared under
routing.signals.response_jailbreak and evaluated from those rules before
any plugin runs, so the observation exists whether or not an enforcement
plugin is enabled. It is published to SignalConfidences and SignalErrors
where every other signal publishes, with detected, not-detected and
unavailable all distinguishable.

The plugin no longer classifies when rules are declared; it consumes the
observation and applies its action, with its own threshold reported as
ignored rather than silently dropped. A configuration that has not
declared rules keeps the old path unchanged.

A decision reading a response-stage signal is not evaluated while the
request is still being routed, since that signal does not exist yet.
This is kept separate from on_unknown, which means "evaluated and could
not resolve"; treating a stage that has not happened as unknown would
let on_unknown: match route a request on a detector that never ran.

Signed-off-by: subin9 <101092510+subin9@users.noreply.github.com>

* [Feature] Put the response stage on the jailbreak rule and evaluate response-stage decisions

Review follow-up for the response-stage signal split.

The stage is now a field on the existing jailbreak signal: a rule with
direction: response scores the model's output, the default request scores
the prompt. There is no new signal type, so the jailbreak:<name> key, the
{type, name} condition, Router Replay and the matched-signal headers carry
the response observation unchanged, and the CLI schema, Dashboard and DSL
compiler/decompiler preserve it as one more rule field. A response-direction
rule rejects method: contrastive, the pattern lists and include_history.

Response-stage decisions are evaluated once the response exists. The
request-stage evaluation still skips them; after the response-direction
rules are scored, the decisions that read them are resolved from the
request-stage matches plus the response observation, beside the request-time
selection rather than replacing it. The response_jailbreak plugin enforces on
whichever decision matched, so a request signal composed with the response
observation drives the response action.

The decision engine is untouched: the stage filter lives in the classifier's
decision evaluation, which is the only place the engine is built.

Tests: the seam is pinned at the filter level with the http_classify stub
(request decision without a plugin, response-stage decision blocks with 403),
a guardrail backend returning 500 is recorded under the rule's key with no
score and fails closed under on_error: block while the default policy
delivers with the failure still on record, and the DSL round-trips direction
in both directions. The response-jailbreak E2E profile gains a composed
response-stage decision case and a streamed-response pass-through case, and
is selected on PRs touching the response filter.

Signed-off-by: subin9 <101092510+subin9@users.noreply.github.com>

* [Feature] Send the compose probe the keyword signal matches and split the direction round-trip test

The compose-block E2E case sent `vsr-response-compose-probe`, but the
profile's keyword signal is `__response_compose_probe__` and keyword rules
match literally, so the request fell through to default_decision and the
response-stage decision was never in play. Same defect 17566cb fixed for the
window cases; this rebase picks that fix up too.

`TestJailbreakDirectionRoundTrip` tripped the cyclop gate at 15 against a
limit of 12. It is now three tests over one compile helper: the compiled
directions, the text round trip, and the AST decompile.

Signed-off-by: subin9 <101092510+subin9@users.noreply.github.com>

* [Feature] Keep the dashboard jailbreak field out of the oversized signal functions

Rebasing onto main picks up the TypeScript structure gate, which
ratchets functions that already exceed 100 lines: they may not grow.
The direction field grew five of them - getSignalFieldSchema,
buildSignalFormFields, and ConfigPageSignalsSection with its view,
editor and save handlers - so the agent lint went red.

Move the jailbreak-specific pieces those functions carry into
module-level helpers: the jailbreak field schema and form fields become
constants the switch and the field list reference, and the section
component reads its jailbreak details, form defaults, edit state and
saved entry from small functions. Every ratcheted function is now
shorter than its baseline, and the field order, labels and behaviour
are unchanged.

Signed-off-by: subin9 <101092510+subin9@users.noreply.github.com>
Claude-Session: https://claude.ai/code/session_01AE8dtxTSjVKP3D3dBG7sgp

* [Feature] Let the selected decision's plugin consume the response signal

The response observation was fed back through the decision engine after
the model answered, and a second decision could be selected from it. That
is not the lifecycle vllm-project#3272 asks for: the request-time selection has
executed, and the response-stage observation belongs to the plugin of the
decision that was selected. Drop EvaluateResponseStageDecision, the
staged decision filter and the response-stage decision fields; the
response_jailbreak plugin enforces on ctx.VSRSelectedDecision only.

A decision rule that names a response-direction rule is rejected at load.
Decisions are selected before the model has answered, so the rule could
only ever read as unknown there, and skipping such decisions at request
time left them unable to match at all.

The rules were read from the root config, which only describes the
default recipe, while classification is recipe-scoped. Read them from
the recipe the request resolved to, through its classifier, so a rule
declared on one entrypoint's recipe scores only that entrypoint's
responses. TestResponseJailbreakSignalReadsTheSelectedRecipeRules is the
named-entrypoint regression: the guarded recipe declares the rule and the
default recipe does not; a response through vllm-sr/guarded is scored and
blocked, one through the default entrypoint is not scored. It fails when
the rules come from the root config.

The compose-block E2E case and its decisions go with the second
selection; the streaming pass-through case now runs through the block
decision. Docs, the reference config and the fragment say the plugin
consumes the observation.

Related vllm-project#3272

Signed-off-by: subin9 <101092510+subin9@users.noreply.github.com>

* [Feature] Record the response-stage observation in Router Replay and the debug header

The replay record is written while the request is routed, before the
model has answered, so the response-direction rule's match, score and
failure never reached it, and the record's response_jailbreak fields
were only ever set at creation. Append the observation as one outcome
per response-direction rule instead: the verdict (detected,
not_detected, unavailable), the score it thresholded or the failure
code, and the action the selected decision's plugin applied. Outcomes
are the append-only post-route channel every store implements, so no
store schema changes. Recorded before a block returns, so a blocked
response leaves the same evidence as a delivered one.

x-vsr-matched-jailbreak was written in the response headers phase, from
the request-stage matches, before the body existed. Rewrite it in the
body phase with the response-direction matches after the request ones,
under the same debug gate as the request-stage signal headers. The
warning E2E case asserts it.

A decision whose response_jailbreak plugin runs with no
response-direction rule declared is reported once at load as the
compatibility path, replacing the per-request debug line.

Related vllm-project#3272

Signed-off-by: subin9 <101092510+subin9@users.noreply.github.com>

* [CI/Build] Tidy dashboard/backend go.sum for the router module's Router Replay store graph

dashboard/backend replaces the router module in, and since vllm-project#3445 on main
`go mod tidy` there wants the checksums of the modules the Router Replay
store pulls in (lib/pq, the Milvus and Qdrant clients, cockroachdb/errors and
their transitive go.mod files). The dashboard job is skipped on main pushes,
so it surfaced on this PR's quality and dashboard gates instead. Generated
with `go mod tidy`; the diff is exactly the one the gate printed.

Signed-off-by: subin9 <101092510+subin9@users.noreply.github.com>

* [Bug] Provision the jailbreak classifier for its response-stage consumers

NeedsJailbreakMappingForRouting and the classifier.jailbreak runtime task
keyed only on a decision rule naming a jailbreak signal. A response-direction
rule is consumed by the selected decision's response_jailbreak plugin and is
never named by a decision rule, so once the E2E profile stopped composing one
into a decision nothing loaded the label mapping: the http_classify
prompt_guard refused to build ("label mapping is required for http_classify")
and the router failed at startup. The plugin-only compatibility path had the
same gap on main, where IsJailbreakEnabled stayed false without the mapping
and the plugin silently never ran.

UsesJailbreakClassifierInReachableRouting counts all three consumers over the
request-reachable profiles (a decision reading a request-direction rule, a
declared response-direction rule, an enabled response_jailbreak plugin), and
the mapping loader, the model download gate and the runtime task use it.

Tests: TestNeedsJailbreakMappingForResponseStageConsumers (flat config,
including the disabled-plugin and prompt_guard-off negatives),
TestNeedsJailbreakMappingForResponseStageConsumersFollowsRecipeReachability
(unreachable recipes do not load it; scoped configs agree) and
TestInitializeRuntimeInitializesJailbreakClassifierForResponseStageConsumers
(the initializer runs once for either consumer, the request-stage ones stay
skipped). All three fail on the previous gating.

Signed-off-by: subin9 <101092510+subin9@users.noreply.github.com>

* [Bug] Leave a partly scanned response unresolved instead of clean

CheckForJailbreakRiskWithThreshold and CheckForJailbreakWithThreshold dropped
a chunk's failure as soon as any other chunk was scored, so a response whose
failed chunk was never inspected came back clean, the response path published
the observation as resolved, and the plugin let it through even under
prompt_guard.on_error: block. A match in a scored chunk still counts, as the
request path keeps a match past an unresolved chunk; without one, a partial
scan now returns an error, which the response path publishes as unresolved so
the on_error policy applies. The security classification API and the
plugin-owned detection path share the helpers and change the same way.

Tests: TestCheckForJailbreakErrorsWhenACleanVerdictNeedsAFailedChunk (risk
and argmax wrappers) and TestResponseJailbreakPartialScanFailureIsNotClean,
whose stub fails every attempt for the chunk carrying a marker (the connector
retries a 5xx) and scores the rest: clean chunks leave the rule unresolved and
on_error: block returns 403, while a match in a scored chunk is still a
detection. Both fail on the previous wrappers.

Signed-off-by: subin9 <101092510+subin9@users.noreply.github.com>

* [Bug] Reject a response-direction rule read through a projection

validateDecisionsReadRequestStageSignals walked only the decision's own rule
leaves. A projection score could still take a response-direction jailbreak
rule as an input, and at request time, before the model has answered, the
absent result became the input's configured miss value, so the decision was
shaped by a signal that could not exist yet. decisionReadsResponseSignal now
follows a projection leaf through its mapping to the score and its inputs,
recursing through projection inputs (a confidence-valued one through the
output it names), and the error names the projection output the rule was read
through.

Tests: TestValidateRejectsProjectionReadingResponseDirectionRule covers a
direct score input, a score of scores and a confidence-valued output, and
accepts the same graph over a request-direction rule; it fails on the
leaf-only walk.

Signed-off-by: subin9 <101092510+subin9@users.noreply.github.com>

* [Bug] Resolve a partly scanned response per jailbreak rule threshold

The response stage scanned the answer once at the lowest declared threshold,
and a match at that threshold dropped the chunk failure. With rules at 0.4
and 0.9, a 0.5 score next to a failed chunk marked the 0.9 rule clean. The
scan now reports the score and the chunk failure apart from each other, and
each rule resolves that one scan on its own line. A rule the score matches
stays a match, and a rule the score does not reach is unresolved while a
chunk is missing.

TestResponseJailbreakPartialScanIsResolvedPerRuleThreshold covers the two
rule case and fails on the previous path.

Related vllm-project#3272

Signed-off-by: subin9 <101092510+subin9@users.noreply.github.com>

---------

Signed-off-by: subin9 <101092510+subin9@users.noreply.github.com>
Signed-off-by: alanxtl <m134679102365478@163.com>